Image formats and related methods and apparatuses

ABSTRACT

Image data may be color graded, distributed and viewed on target displays. Mappings that preserve mid-range points and mid-range contrast may be applied to view the image data for color grading and to prepare the image data for display on a target display. The image data may be expanded to exploit the dynamic range of the target display without affecting mid-tone values.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Application No.61/539,438, filed Sep. 26, 2011, hereby incorporated by reference in itsentirety.

TECHNICAL FIELD OF THE INVENTION

The invention relates to the media industry and has application indelivery of image content such as video and/or still images to consumerswho view the images on displays such as televisions, digital cinemadisplays, media viewers and the like. The invention has particularapplication to formats for transmission of image data, displays, andmethods for encoding, decoding and/or displaying image data.

BACKGROUND OF THE INVENTION

Video content is typically prepared and distributed by way of acombination of systems and technologies that may be termed a “videodelivery pipeline”. FIG. 1 is a flowchart of a conventional videodelivery pipeline 100 showing various stages from video capture to videocontent display. A sequence of video frames 101 is captured at block102. Video frames 101 may be digitally captured (e.g. by a digitalcamera) or generated by a computer (e.g. using computer animation) toprovide video data 103. Alternately, video frames 101 may be captured onfilm by a film camera. The film is converted to a digital format toprovide video data 103. In a production phase 104 video data 103 isedited to provide a video production 105.

Video data of production 105 is provided to a processor at block 106 forpost-production editing. Block 106 post-production editing may includeadjusting or modifying colors or brightness in particular areas of animage to enhance the image quality or achieve a particular appearancefor the image in accordance with the video creator's creative intent.This is sometimes called “color timing”. Other editing (e.g. sceneselection and sequencing, image cropping, addition of computer-generatedvisual special effects, etc.) may be performed at block 106 to yield afinal version 107 of the production for distribution. During block 106post-production editing, video images are viewed on a reference display108. In block 106 the final production 107 is viewed on referencedisplay 108 or another reference display for approval. It is notmandatory that the same display be used for color timing and approval.

Following post-production, video data of final production 107 isdelivered at block 116 to a display subsystem 120. As seen in FIG. 1A,block 116 delivery includes an encoder stage 117A which generatesencoded video data 118 embodying the content of video data 107 to bedistributed by way of a video distribution medium 115 (e.g. satellite,cable, DVD, wireless communication link, internet, local area network,broadcast, etc.). A decoder stage 117B is located downstream fromencoder stage 117A to decode video data 118 transmitted over medium 115.

Display subsystem 120 may perform video processing 120A and displaying120B. Video processor 120A may be integrated with displaying 120B or maybe separate. At block 120A, video data 118 is provided to a videoprocessor for processing and/or decoding. Video data 118 is output to adisplay 122 at block 120B to display a sequence of images to a viewer.

Encoded video data 118 may have a format selected with reference toproperties of medium 115 (for example to fit within bandwidthrequirements and/or format requirements of medium 115). To improve thequality of displayed images, encoded video data 118 may be driventhrough video delivery pipeline 100 at a relatively high bit rate so asto facilitate an increased bit depth for defining RGB or chroma valuesfor each chrominance (color) channel. For example, video data 118 maycomprise 8, 10 or 12 bits of data for each color channel of a pixel. Thevideo data may be compressed.

Despite using a high bit depth for each chrominance channel, variationsin display characteristics (such as luminance range, gamut, etc.) mayaffect the appearance of an image rendered on a display so that theimage rendered does not match the creative intent of the video'screator. In particular, the perceived color or brightness of an imagerendered on a particular display subsystem may differ from the color orbrightness of the image as viewed on reference display 108 duringpost-production block 106.

The same video content 107 may be displayed on any of a wide variety ofdifferent types of electronic displays including televisions, computerdisplays, special purpose displays such as medical imaging displays orvirtual reality displays, video game displays, advertising displays,displays on cellular telephones, tablets, media player displays,displays in hand-held devices, displays used on control panels forequipment of different kinds and the like. Displays may employ any of awide range of technologies. Some non-limiting examples are plasmadisplays, liquid crystal displays (LCDs), cathode ray tube (CRT)displays, organic light emitting diode (OLED) displays, projectiondisplays that use any of various light sources in combination withvarious spatial light modulation technologies, and so on.

Different displays may vary significantly with respect to features suchas:

-   -   the color gamut that can be reproduced by the display;    -   the maximum brightness achievable;    -   contrast ratio;    -   resolution;    -   acceptable input signal formats;    -   color depth;    -   white level;    -   black level;    -   white point; and    -   grey steps.        Consequently, the same image content may appear different when        played back on different displays. Image content that matches a        creator's creative intent when displayed on some displays may        depart from the creator's creative intent in one or more ways        when viewed on other displays. The appearance of displayed        images is also affected by the environment in which a display is        being viewed. For example, the luminance of ambient lighting,        the color of ambient lighting and screen reflections can all        affect the appearance of displayed images.

With the increasing availability of high-performance displays (e.g.displays that have high peak luminance and/or broad color gamut) comesthe problem of how to adjust images for optimum viewing on a particulardisplay or type of displays. Addressing this problem in simplistic wayscan result in noticeable artifacts in displayed images. For example,consider the case where an image that appears properly on a displayhaving a moderate peak luminance is displayed on a target display havinga very high peak luminance. If one expands the luminance range of theimage data to take advantage of the high peak luminance of the targetdisplay, the result may be poor due to objectionable artifacts that arerendered apparent by the range expansion. Artifacts may include, forexample, one or more of banding, quantization artifacts, visiblemacroblock edges, objectionable film grain and the like. On the otherhand, if the image is displayed on the target display without rangeexpansion, no benefit is gained from the high peak luminance that thetarget display can achieve.

Video formats used in many current imaging systems (e.g. HDTV, UHDTV)are based on defining black and white levels with a power response. Thismakes it very difficult to ensure consistent midtones, contrast, andcolor when video content derived from the same video data 107 is viewedon different displays.

The system pipeline response can be represented by an end-to-endtransfer function that compares an image on a reference display and thesame image displayed on a target display. Makers of displays typicallymake assumptions about the response of the reference display (andenvironment) in which images were approved. Displays may process imagedata in various ways (e.g. applying a power function response, makingcolor saturation adjustments, adjusting brightness adjustment, adjustingcontrast etc.) to arrive at an image that the display maker thinks willbe best appreciated by viewers.

For accurate reproduction of video images, video distribution systems ofthe type illustrated in FIGS. 1 and 1A typically require that the systemresponse of the pipeline be tailored to match characteristics of thereference display on which the content was approved and/or color graded.Display makers can attempt to provide image processing that achieves adesired system response, for example by defining a response curvebetween a minimum and maximum luminance. The minimum and maximumluminance may be implied from assumed capabilities of a referencedisplay. Displays that perform image processing based on wrongassumptions regarding the characteristics of a reference display onwhich the image data was approved or color graded may produce imagesthat are not faithful to the image as approved by its creator. Forexample different interpretations of the same video signal can result ininconsistent midtones, and other characteristics of images displayed ondifferent displays.

Another issue with video distribution systems of the type illustrated inFIGS. 1 and 1A is that changing to a different reference display oraltering the configuration of the reference display or its viewingenvironment can require changes to the system response of the pipelineif one wishes to ensure that viewers have the highest quality viewingexperience. One approach to dealing with this issue is to providemetadata along with video data 118. The metadata can specify the sourcegamut. The target displays may then be made to adapt themselves byproviding image processing selected based on the metadata to achieve thedesired system response. This adds complexity and is prone to failurefor some distribution channels (e.g. broadcast).

There is a general desire for systems, apparatus and methods forgenerating, delivering, processing and displaying video data to preservethe content creator's creative intent. There is a general desire forsystems, apparatus and methods for providing information which may beused to guide downstream processing and/or display of video data.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection. Similarly, issues identified with respect to one or moreapproaches should not assume to have been recognized in any prior art onthe basis of this section, unless otherwise indicated.

SUMMARY OF THE INVENTION

This invention has several aspects. These include methods for displayingdigital images, methods for preparing digital images for distribution,video delivery pipelines and their respective components.

One aspect of the invention provides image data encoding methods whichfix mid-tones of both reference and target displays to a common responseand allow highlights and shadows to be extended and/or compressed basedon the capabilities of the target display. The result is an imagingsystem in which image processing at the target display can be performeddepending only on characteristics of the target display orcharacteristics of the target display and its environment (and does notneed to be based on characteristics of the image source). In someembodiments image data is assigned to image values represented by imagedata according to a sigmoidal response curve. The same sigmoidalresponse curve may be used in the reference display used in creationand/or approval of the source content.

Another aspect of the invention provides image data encoding methods inwhich the sizes of the steps between levels in which luminance isencoded increase more slowly and/or become smaller as luminanceincreases above a mid-tone range and/or decreases below the mid-tonerange.

Another aspect of the invention provides methods for encoding imagedata. The methods comprise processing image data to provide image valuescorresponding to quantized luminance steps wherein a step size betweenthe quantized luminance steps is smaller in a highlight region than in amid-tone region.

Another aspect of the invention provides methods for displaying images(which may be still or video images) on a target display. The methodscomprise processing image data that has been reviewed on a referencedisplay and encoded such that a predetermined image value of the imagedata corresponds to a predetermined mid-tone value displayed on thereference display. The processing comprising applying a mapping to theimage data that maps the predetermined image value to the predeterminedmid-tone value when viewed on the target display. In certain embodimentsthe mapping provides a log-linear mid-tone region having a slope n thatmatches or approximates a slope of a corresponding mid-tone region of aresponse function of the reference display. In an example embodiment themapping is specified by a sigmoidal function. An example sigmoidalmapping function is given by the equation:

$L_{OUT} = \frac{c_{1} + {c_{2}L_{IN}^{n}}}{1 + {c_{3}L_{IN}^{n}}}$which is discussed below.

Some embodiments involve adjusting the mapping in response to ambientlighting. The adjusting may, for example, comprise one or more of:increasing the mid-tone value to which the predetermined image value ismapped in response to an increase in the ambient lighting; anddecreasing mid-tone contrast of the mapping function.

Some embodiments comprise modeling the effect of the ambient lighting onlight adaptation of the human visual system to obtain an estimatedadaptation and adjusting the mapping based on the estimated adaptation.Modeling the effect of the ambient lighting on the adaptation of thehuman visual system may optionally comprise modeling a plurality ofadaptation aspects to obtain a corresponding plurality of estimatedadaptations. In such cases the method may comprise adjusting differentcharacteristics of the mappings based on different ones of the estimatedadaptations. For example, adjusting the mapping may comprise one or moreof: controlling a luminance to which the predetermined image value ismapped based at least in part on an estimate of a general brightnessadaptation; controlling mid-range contrast based at least in part on anestimate of lateral brightness adaptation; and controlling contrast inhighlight and shadow regions based at least in part on an estimate oflocal brightness adaptation.

In cases where the target display has a greater peak luminance than thereference display the method may comprise applying a mapping configuredto expand the luminance corresponding to image values corresponding to ahighlight region above the mid-tone region. In cases where the targetdisplay has a lower black level than the reference display the methodmay comprise applying a mapping configured to expand the luminancecorresponding to image values corresponding to a shadow region below themid-tone region.

Another aspect of the invention provides cameras and other image capturedevices that map detected light to output image values according to aresponse function. In some embodiments the response function is asigmoidal function. An example image capture device according to thisaspect comprises a light-sensing array operable to detect light and amapping unit connected to represent intensity of light sensed byelements of the light-sensing array as digital image values according toa response function. An exposure compensation mechanism is connected tocontrol the mapping unit such that a fixed point in the mid-range of theresponse function is made to correspond to a midpoint of the outputimage values. In some embodiments the response function is given by theequation:

$L_{OUT} = \frac{c_{1} + {c_{2}L_{IN}^{n}}}{1 + {c_{3}L_{IN}^{n}}}$and the exposure compensation mechanism controls the value of one orboth of n and c₂ based at least in part on a measured exposure.

Other example aspects of the invention provide displays and imageprocessing apparatus.

Further aspects of the invention and features of specific embodiments ofthe invention are described below.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate non-limiting example embodiments ofthe invention.

FIG. 1 is a block diagram of a video distribution pipeline.

FIG. 1A is a block diagram illustrating distribution of image contentover a distribution medium.

FIG. 2 is a graph which illustrates a possible gamma-based responsecurve for a reference display.

FIG. 2A is a graph which illustrates possible response curves for twodifferent example target displays.

FIG. 2B is a graph which illustrates system response curves for the casewhere video data approved on a display having a response as shown inFIG. 2 is displayed on two different target displays having the responsefunctions of FIG. 2A.

FIG. 3 shows a response curve for an example reference display.

FIG. 3A shows response curves corresponding to three example targetdisplays.

FIG. 3B shows curves representing a system response for differentcombinations of two different target displays and two differentreference displays having different peak luminance.

FIG. 4 is a block diagram for an example video delivery pipeline.

FIG. 5 shows example response curves that may be provided by a mappingunit for dim and bright ambient conditions.

FIG. 5A shows system response curves corresponding to the responsecurves of FIG. 5.

FIG. 6 shows possible response functions that may be provided in capturedevices for different ambient lighting conditions.

FIG. 7 shows example system response curves corresponding to the curvesof FIG. 6.

DESCRIPTION OF THE INVENTION

Throughout the following description, specific details are set forth inorder to provide a more thorough understanding of the invention.However, the invention may be practiced without these particulars. Inother instances, well known elements have not been shown or described indetail to avoid unnecessarily obscuring the invention. Accordingly, thespecification and drawings are to be regarded in an illustrative, ratherthan a restrictive, sense.

Luminance and other values in image data (“image values”) are typicallygamma encoded. Simple gamma encoding involves applying a power law toimage data, for example according to:T _(OUT) =AT _(IN) ^(γ)where T_(IN) is the input value for luminance or other image value,T_(OUT) is the output value for the luminance or other image value, A isa constant (typically 1) and γ is the gamma value. Images are typicallyencoded with values for γ that are less than one (for example, about0.45). When gamma-encoded images are displayed, the display processapplies a gamma expansion (gamma value larger than 1—for example 2.2 or2.4) which results in a desired image being displayed.

To date, standard dynamic range (SDR) reference displays have powerfunction response characteristics. The inventors have determined thatthis response does not preserve end-to-end system luminance levels whencontent is shown on displays having luminance ranges different from thatof the reference display. This distorts image appearance. FIG. 2 shows ageneralized example response curve 201 for an example reference display.FIG. 2A shows example response curves 202A and 202B for two differenttarget displays. All three displays have the same gamma (2.4 in thisexample) but the target displays have higher peak luminance than thereference display and the target display corresponding to curve 202B hasa higher black level than the reference display.

FIG. 2B shows the resulting system response curves 203A and 203B forimages respectively viewed on the two example target displays havingresponse curves 202A and 202B. For a perfect match of appearance (theimage would appear the same on the target display as on the referencedisplay) luminance levels would fall on dashed line 204. However, sinceboth target displays have a higher peak luminance than the referencedisplay, the output levels far exceed the intent. If the visual systemsof viewers viewing the target displays are adapted to the same ambientlighting conditions as for the reference display, the result would be animage which appeared much brighter than intended. Also, system responses203A and 203B are altered near black. Response curve 203A issubstantially steeper in the dark region 205 than the ideal responsecurve 204. This has the effect of stretching levels apart, or increasingcontrast, further distorting the creative intent.

Curves 206A and 206B in FIG. 2B are response curves for the case wherethe reference display has a higher peak luminance (e.g. 600 nits asopposed to 100 nits) but with the same gamma=2.4 response. These curvesshow how the system response curves change drastically throughout theentire range depending on the characteristics of the reference display,indicating that appearance is dependent on the characteristics of thereference monitor. FIGS. 2 through 2B demonstrate that mismatch betweenthe capabilities of the reference and target displays in a gamma-basedsystem result in appearance not being preserved through the videopipeline. This can degrade the viewing experience.

The inventors have determined that at least some of the problems withthe prior art approach described above arise because values in encodedvideo data do not correspond to specific luminance values but instead tovalues determined with reference to the ranges of the displays on whichthey are displayed. This is a special problem with mid-range values(e.g. values in the range of about 0.1 nits to 100 nits in a dim viewingenvironment). Pixels have mid-range values in the most salient portionsof most images. This issue may be addressed by providing image dataformats and systems in which mid-range values are consistently mappedinto the same range.

One way to achieve this is illustrated in FIGS. 3 and 3A. These figuresillustrate response curves for a number of displays that are constructedsuch that mid-range values in image data are consistently mapped intothe same luminance range. In these example curves, values outside of themid-range are compressed or expanded smoothly and gently into the rangeof the display. FIG. 3 shows a response curve 301 corresponding to anexample reference display. FIG. 3A shows response curves 302A, 302B, and302C corresponding to three example target displays. In each case, forimage values in a mid-range region 303 the mappings of image values toluminance is essentially the same.

A feature of response curve 301 is that for higher image values theluminances increase relatively slowly with increasing values (theluminances increase in relatively small steps). This results inredundant precision at such higher values (since the human visual systemcannot distinguish between luminances that differ by less than athreshold amount). This, in turn, facilitates expansion of the luminancecorresponding to higher image values without creating perceptiblequantization artifacts such as banding and the like that can result whenexpansion causes adjacent image values to correspond to noticeablydifferent luminance levels.

In some embodiments, response curve 301 is such that the difference inluminance specified by adjacent levels is smaller than a just noticeabledifference (JND) at least in the portion of response curve 301 towardhigher luminance values (for example, at least in the right hand ¼ ofcurve 301). There are various ways to quantify the JND. A simple way toquantify JND is Weber's ratio which can be given by:

$R = \frac{\Delta\; L}{L}$where ΔL is the luminance step size and L is the background luminancelevel. One estimate of the JND is that the JND corresponds to a Weber'sratio of 1%. In some embodiments, the response function 301 for areference display is such that the difference in luminance specified byadjacent image values is less than ½, ⅕ or 1/10 of a JND. This permitsexpansion of the luminance range by a factor of up to 2, 5 or 10 whilepreserving the desirable quality that the difference in luminancespecified by adjacent quantized image values does not exceed a JND. Forexample, adjacent image values may correspond to adjacent luminances forwhich the Weber's ratio is less than 0.005, 0.002 or 0.001. In otherembodiments another estimate of JND is applied. for example, asdescribed in G. Ward, A Contrast-Based Scalefactor for LuminanceDisplay, Graphics Gems IV, Ed. by P. S. Heckbert, pp. 415-421, 1994 orJ. A. Ferwerda, et al. A Model of Visual Adaptation for Realistic ImageSynthesis, Computer Graphics, pp. 249-258, 1996. In such cases, theresponse curve 301 for a reference display may be such that thedifference in luminance specified by adjacent image values is less than½, ⅕ or 1/10 of a JND to provide for the possibility of a desired rangeof expansion without causing artifacts which result from adjacentquantized luminance values differing by significantly more than a JNDwhen displayed on a target display.

In the embodiment illustrated in FIGS. 3 and 3A the response functionsof the reference and target displays are all sigmoidal functions. Ineach case, the sigmoidal function comprises parameters that allow afixed point to be set. The fixed point is a point corresponding to aspecific mid-range image value and a specific corresponding luminancevalue. The sigmoidal function also has one or more parameters that allowcontrol over the slope of the response function in a mid-range regionthat includes the fixed point. These parameters can be set so that theresponse functions for the reference and target displays all passthrough the fixed point and all have the same slope in a mid-rangeregion.

The sigmoidal response functions may differ in the ends of the range ofimage values. This flexibility allows the ends of the response functionsto be arranged to extend to the display's capabilities. In thisillustrated example embodiment, on all displays the input image valuesare in a range of zero to one. In each case, the input image value 0.5maps to 10 nits with the same mid-range slope (contrast). The ends ofeach response function smoothly roll off according to the display'scapabilities.

The response functions of target displays may be specified as astandard. For example, a sigmoidal response may be specified in such amanner that the shape of the response curve is specified by parametersthat specify a minimum, a maximum, a mid-range value and a mid-rangeslope (or contrast). The standard may specify the mid-range value and amid-range slope. The reference display and target displays may all useresponse curves characterized by the same mid-range value and amid-range slope. Different target displays may use different maximum andminimum values. The response curves in the target displays may smoothlyroll off to the maximum and minimum values.

A specification does not necessarily fix a mid-point in an absolutesense. As an alternative, a specification could specify a rule fordetermining the mid-point based on capabilities of the target display.For example the mid-point may be specified based on a geometric meanluminance of the target display. In an example, the mid point may be setfor a target display by determining an intermediate value Lat′ accordingto:L _(at′) =A√{square root over ((L _(min) ·L _(max)))}where Lmin and Lmax are the extremes of the range of the target displayand A is a constant and then determining the mid-point for the targetdisplay according to:L _(at)=√{square root over (L _(as) ·L _(at′))}where Las is a predetermined midpoint (e.g. 10 nits) and Lat is themid-point to be used for the target display.

Similarly, mid-range contrast need not be specified as a fixed value butcould, in some embodiments, be based in part on capabilities of a targetdisplay. For example, mid-range contrast could be based on a value n′determined according to:

$n^{\prime} = {\frac{1}{2}{\log_{10}\left( \frac{L_{\max}}{L_{at}} \right)}}$the mid-range slope, n, for the response curve of the target display maythen be given by:n=√{square root over (n _(S) ·n′)}where ns is a specified standard slope.

Roll off of the response functions for a reference or target display athigh image values may be determined based on the peak luminance of thedisplay. A display with 120 nit peak luminance will have a much moreshallow rolloff than a display with 1200 peak luminance. The exact slopeat any point in the rolloff region for a reference display is not tooimportant from the point of view of a colorist as the colorist willcompensate for it during mapping.

FIG. 3B shows curves 310A and 310B representing the system response fortwo different target displays for the case where the reference displayhas a maximum luminance of 600 nits and curves 311A and 311B for thecase where the reference display has a maximum luminance of 5000 nits.It can be seen that the mid-range is not affected by these changes andthe system response in the mid-range is consistent across the targetdisplays. The top and bottom ends of the system response functionsstretch to exploit the capabilities of the target displays.

In an example embodiment, the response curves for the displays are givenby:

$L_{OUT} = \frac{c_{1} + {c_{2}L_{IN}^{n}}}{1 + {c_{3}L_{IN}^{n}}}$where L_(IN) is the input value for luminance or other image value. Forexample, L_(IN) may be a luminance in a range [min_(S), max_(S)] wheremin_(S) and max_(S) are respectively standard minimum and maximumvalues. L_(OUT) is the output value for the luminance or other imagevalue, c₁, c₂ and c₃ and n are parameters. Here, the value of n sets themid-range contrast and the location of the mid-range fixed point isdetermined primarily by the value of c₂. The parameters c₁ and c₃ may beused to tune the response function to exploit the full dynamic range ofa display.

The image values present in image data may be further encoded. In anexample embodiment, the image values represent the logarithms ofluminance levels. In an example of such a case, L_(IN) may be given inthe range of [0.001, 1000] by:L _(IN)=10^((6V-3))where V is a normalized image value in the range of zero to one.

As noted above, response functions may be determined with reference to astandard range [min_(S), max_(S)]. It is possible to select the standardrange to provide some degree of backward compatibility with videosignals that are gamma encoded. For example, video data representedusing a sigmoidal response function as defined above can be made todisplay nearly correctly on existing displays which assume standardgamma-encoded video input by appropriate selection of the standardmid-point, standard mid-range slope and standard range. For example,selecting [min_(S), max_(S)] to be [0.005, 120] with suitable choicesfor ns and Las can yield a video signal that can be viewed on typicaltelevisions with acceptable results.

Response functions having properties like that of the example responsefunction of Equation (2) may be applied in various ways to provide mediadelivery pipelines. One example video delivery pipeline 400 isillustrated in FIG. 4. Pipeline 400 processes and distributes videocontent based on source video 402. A color grading station 404 includesa reference display 406. A mapping unit 410 maps color graded video 405to be viewed on reference display 406. Mapping unit 410 may, forexample, implement a mapping function of the type illustrated byEquation (2). Mapping unit 410 may be a stand-alone device or may beintegrated with reference display 406, color grading station 404, someother device, or a combination of these.

A color grader can use controls of color grading station 404 to adjustimage values from source video 402 to yield color graded video 405. Thecolor grader can view color graded video 405 on reference display 406and continue to make adjustments to color graded video 405 usingcontrols provided by color grading station 404 until the color gradedvideo 405—as mapped by mapping unit 410—has a desired appearance whenviewed on reference display 406.

Color graded video 405 may then be distributed over a distributionchannel 412 for viewing on one or more target displays 420. The colorgraded video 405 may be suitably encoded and decoded in distributionchannel 412. In some embodiments the color graded video 405 isquantized, encoded and represented in a data format apropos to theparticular distribution channel 412 being used. In some embodimentscolor graded video 405 is formatted according to a VDR (visual dynamicrange) format in distribution channel 412. VDR format is a video formatdescribed in commonly assigned PCT Application No. PCT/US2010/022700entitled “EXTENDED DYNAMIC RANGE AND EXTENDED DIMENSIONALITY IMAGESIGNAL CONVERSION AND/OR DELIVERY VIA LEGACY VIDEO INTERFACES” which ishereby incorporated herein by reference for all purposes.

Each target display 420 is associated with a mapping unit 422 that mapscolor graded video 405 for display on the target display 420. Mappingunits 422 may be integrated with target displays 420 or a suitablemapping unit 422 may be otherwise provided upstream from each targetdisplays 420.

Mapping units 422 do not require information such as a reference gamutor environmental conditions regarding reference display 406 because themappings performed by mapping units 410 and 422 have a common fixedmid-point and mid-tone contrast so that the mid-point and mid-tonecontrast of images viewed on target displays 420 are automatically thesame as those of the images when viewed on reference display 406.

A mapping having parameters that allow control over the luminance that aspecific mid-point will be mapped to and control over mid-range contrast(as, for example are provided by the parameters c2 and n of Equation(2)) can be applied to advantage in adjusting displays to account forambient viewing conditions.

The human visual system reacts differently to images depending on itsadaptation to light. For example, a viewer will perceive the same imageson a television differently depending on whether the television is beingviewed in an otherwise dark room or in a brightly lit room. Lateraladaptation describes the adaptation of a viewer's visual system to thebrightness of the environment (including a display and itssurroundings). Chromatic adaptation describes the adaptation of aviewer's visual system to the chromaticity of the viewer's surroundings(for example the color temperature of ambient lighting). Anotherenvironmental effect that can alter a viewer's perception of images on adisplay is viewing flare. Viewing flare describes the effect of screenreflections.

In general, lateral adaptation to brighter levels has the effect ofraising what is perceived as “mid grey”, and also the perceivedcontrast. Lateral adaptation may be compensated for by raising themidpoint and lowering the mid-tone contrast. In some embodiments,mapping units 422 receive input from an ambient light sensor 425 and areconfigured to increase the luminance to which the mid-point image valueis mapped (for example, by adjusting the value of c₂ in a mappingfunction as provided by Equation (2)) and/or to decrease mid-tonecontrast (for example, by adjusting the value of n in Equation (2)) inresponse to the input from ambient light sensor 425 indicating greaterambient light intensity.

Viewing flare has the effect of raising black level due to reflectionsof ambient light from a display screen. The result is typically areduced dynamic range of the display and a crushing of dark detail.Viewing flare typically raises the level of the deepest black but doesthe level of the peak white is mostly unaffected. Viewing flare can alsocause desaturation of dark colors. Viewing flare can be compensated forby increasing the contrast in dark regions affected by the reflections.In some embodiments, mapping units 422 alter the mapping of dark levelsin response to input from ambient light sensor 425 to increase contrastin dark regions in response to increases in ambient light. In someembodiments light sensor 425 is oriented to selectively detect lightdirected toward a screen of display 420 or a separate light sensor 425Ais provided that selectively detects light directed toward the screen ofdisplay 420. In such embodiments, the boost in dark range contrast maybe based on a measure of the ambient light directed toward the screen ofdisplay 420. The function used to boost dark contrast advantageously hasan effect that tapers of with increasing image values such thatmid-tones and highlights are not significantly affected. In anembodiment which applies the mapping function of Equation (2), increasein dark-range contrast may be achieved by adjusting the parameter c₁.Such a function may be implemented in various alternative ways includingby pre-processing image values to increase dark level contrast prior tothe normal mapping performed by mapping unit 422 or post-processingafter the normal mapping performed by mapping unit 422 or byimplementing an alternative mapping in mapping unit 422 that has one ormore parameters permitting control of dark-range contrast.

FIG. 5 shows example response curves 501A and 501B that may be providedby a mapping unit 422 for dim and bright ambient conditionsrespectively. FIG. 5A shows corresponding system response curves 502Aand 502B. In FIG. 5A, line 503A indicates an appearance match betweenthe reference and target displays for the case of dim ambient conditionsand line 503B indicates an appearance match between the reference andtarget displays for the case of bright ambient conditions.

Some embodiments estimate reflections of ambient light from a screen ofdisplay 420. Such reflections may be estimated from measurements of theambient light by sensor(s) 425 and/or 425A and known opticalcharacteristics of the display screen. In some embodiments a signalrepresenting measured ambient light is multiplied by a factor which isdetermined empirically or based on knowledge of the opticalcharacteristics of the display screen to obtain an estimate of reflectedlight that is added to the luminance created by the display of images onthe display 420.

Adaptation of the human visual system to light may be estimated byinputting information about the history of light exposure to amathematical model of the behavior of the human visual system. In someembodiments, mapping units 422 implement algorithms that apply suchmathematical models using as inputs values received from one or moreambient light sensors 425 and/or information regarding the brightness ofcontent displayed on display 420. The modeled adaptation will, ingeneral be a function of past ambient conditions. The modeled adaptationmay take into account light emitted by a display instead of or as wellas other ambient light at a viewing location. In some embodiments,ambient light may be estimated based at least in part on the video dataoutput by a mapping unit 422. In some embodiments, mapping units 422implement methods and apparatus as described in U.S. application No.61/433,454 filed on 17 Jan. 2011 and entitled “METHODS AND APPARATUS FORESTIMATING LIGHT ADAPTATION LEVELS OF PERSONS VIEWING DISPLAYS” which ishereby incorporated herein by reference for all purposes.

Light adaptation has a number of different aspects. In some embodiments,control over different aspects of the mapping performed by mapping units422 is based on estimates of different adaptation aspects. For example,control over the luminance to which a mid-point image value is mappedmay be based on an estimate of a general brightness adaptation. Thegeneral brightness adaptation may be based on the average brightness towhich the viewer has been exposed over a time period characteristic ofthe general adaptation of the human visual system. Control overmid-range contrast may be based on an estimate of lateral brightnessadaptation. Mapping unit 422 may apply a plurality of models to estimatea corresponding plurality of different types of adaptation and maycontrol a corresponding plurality of different aspects of the mappingbased on outputs of the plurality of models.

A mapping unit 422 may have any of a wide variety of constructions andmay be implemented in software and/or hardware. Mapping units 422 mayimplement features as described in the commonly-assigned U.S.application Nos. 61/453,107 filed on 15 Mar. 2011 and entitled “METHODSAND APPARATUS FOR IMAGE DATA TRANSFORMATION” and/or 61/473,691 filed on8 Apr. 2011 and entitled “IMAGE RANGE EXPANSION CONTROL METHODS ANDAPPARATUS” which are both hereby incorporated herein by reference forall purposes.

It can be appreciated that some embodiments provide a content deliverysystem in which image data distributed by the system is tailored forviewing on a “virtual display” having specified capabilities. Thecapabilities of the virtual display are specified in advance. When thecontent is to be viewed on a particular physical display (either areference display or a target display), the content may be mapped asdescribed herein in a manner which ensures that mid-range contrast and amid-point value are displayed consistently while dark values and brightvalues are displayed according to the capabilities of the physicaldisplay. The distributed data may have redundant precision (i.e. greaterprecision than is required for displaying the data on a referencedisplay) at least at values corresponding to brightness above amid-range. The mappings may all have a specified functional form (e.g. asigmoidal form). In such a system the approved appearance of distributedcontent may be preserved across a wide range of target displays.

For broadcast and other applications, a mapping as performed by mappingunit 410 may be provided in a camera or other image capture device. Forexample, a mapping unit 410 that implements a mapping according toEquation (2) may be associated with a camera. Some or all of theparameters c1, c2, c3, and n may be adjusted manually or automaticallyto yield image data that provides a desired image on target display 420.The mapping may be adjusted to exploit the limited dynamic range andprecision of the light sensor(s) in the capture device under thelighting conditions that the capture device is experiencing. FIG. 6 is aplot of possible response functions that may be provided in capturedevices for different ambient lighting conditions. Curve 601A is anexample response function for a capture device having a relatively lowdynamic range operating to capture a dimly lit scene. Curve 601B is anexample response function for the same capture device operating tocapture a brightly lit scene. Curve 602 is an example response functionfor a capture device having a relatively high dynamic range operating tocapture a dimly lit scene.

The gain and aperture on the capture device operate to shift theresponse curves curve up or down such that a fixed point in themid-range of the curve is made to correspond to a midpoint of the outputimage values. The contrast is adjusted depending on the exposure.

It can be seen that curve 602 provides markedly increased contrast forextreme regions 603A and 603B of its dynamic range. These regions may berespectively called a dark end region and a bright end region. Suchranges may be outside of the range that can be realistically displayedby most target displays. Providing increased contrast in ranges 603A and603B results in relatively more signal values being allocated to rangesthat can be reproduced by typical displays and relatively fewer signalvalues being allocated to ranges 603A and 603B. This means thatluminance values in regions 603A and 603B will be compressed whendisplayed on most target displays.

Where a capture device implements a response function as describedabove, the capture device may be operated to encode linear light levelsinto image values such that the midpoint and contrast are correct whenprocessed by a mapping unit 422 and displayed on a display 420 withoutfurther processing. The overall system response from the combination ofprocessing at the image capture device and at the display may:

-   -   cause the middle of the exposure of the captured scene to be        displayed in the middle of the display's dynamic range;    -   allow for slight changes in mid-tone contrast to account for        different viewing environments; and    -   fit the full captured range into the dynamic range of display        420 without further clipping of darks or brights.        FIG. 7 shows some example system response curves 701A, 701B, and        702 that correspond respectively to curves 601A, 601B, and 602        of FIG. 6. In FIG. 7, the “dim” environment corresponded to the        center of the response curves at 10 nits. When displaying in the        same environment, we see that the appearance is preserved by        displaying this same portion at 10 nits. Curve 702B shows that        in an image captured in a brighter environment with a mid-tone        value at 100 nits, the mid-tone value gets mapped to 10 nits on        the with a corresponding increase in contrast to account for the        dimmer environment. Curves 701A and 702 show that, when operated        to capture images in a dim environment, both standard dynamic        range (SDR) and high dynamic range (HDR) sensors map the        midrange correctly and smoothly roll-off in black and white        according to the combined capabilities of the camera or other        image capture device and display.

Certain implementations of the invention comprise computer processorswhich execute software instructions which cause the processors toperform a method of the invention. For example, one or more processorsmay implement methods as described herein by executing softwareinstructions in a program memory accessible to the processors. Theinvention may also be provided in the form of a program product. Theprogram product may comprise any non-transitory medium which carries aset of computer-readable signals comprising instructions which, whenexecuted by a data processor, cause the data processor to execute amethod of the invention. Program products according to the invention maybe in any of a wide variety of forms. The program product may comprise,for example, physical media such as magnetic data storage mediaincluding floppy diskettes, hard disk drives, optical data storage mediaincluding CD ROMs, DVDs, electronic data storage media including ROMs,flash RAM, or the like. The computer-readable signals on the programproduct may optionally be compressed or encrypted.

Where a component (e.g. a software module, processor, assembly, device,circuit, etc.) is referred to above, unless otherwise indicated,reference to that component (including a reference to a “means”) shouldbe interpreted as including as equivalents of that component anycomponent which performs the function of the described component (i.e.,that is functionally equivalent), including components which are notstructurally equivalent to the disclosed structure which performs thefunction in the illustrated exemplary embodiments of the invention.

As will be apparent to those skilled in the art in the light of theforegoing disclosure, many alterations and modifications are possible inthe practice of this invention without departing from the scope thereof.

What is claimed is:
 1. A method for displaying an image on a targetdisplay, the method comprising: processing image data that has beenreviewed on a reference display and encoded such that a predeterminedimage value of the image data corresponds to a predetermined mid-tonevalue displayed on the reference display, the processing comprisingapplying a mapping to the image data that maps the predetermined imagevalue to the predetermined mid-tone value when viewed on the targetdisplay, wherein a dynamic range of the target display differs from adynamic range of the reference display; wherein the mapping sets theslope of the mid-tone region of a response function when viewed on thetarget display to match a corresponding mid tone region of a responsefunction of the reference display; and wherein the mapping comprises amapping function given by:$L_{OUT} = \frac{c_{1} + {c_{2}L_{IN}^{n}}}{1 + {c_{3}L_{IN}^{n}}}$where L_(OUT) is an output of the mapping L_(IN) is an input of themapping and c₁, c₂, c₃ and n are numeric parameters.
 2. A methodaccording to claim 1 comprising adjusting the mapping in response toambient lighting.
 3. A method according to claim 2 wherein adjusting themapping comprises increasing the mid-tone value to which thepredetermined image value is mapped in response to an increase in theambient lighting.
 4. A method according to claim 3 wherein increasingthe mid-tone value comprises adjusting the value of c₂ in the mappingfunction.
 5. A method according to claim 2 wherein adjusting the mappingcomprises decreasing mid-tone contrast of the mapping function.
 6. Amethod according to claim 5 wherein decreasing the mid-tone contrastcomprises adjusting the value of n.
 7. A method according to claim 2comprising modeling the effect of the ambient lighting on lightadaptation of the human visual system to obtain an estimated adaptationand adjusting the mapping based on the estimated adaptation.
 8. A methodaccording to claim 7 wherein modeling the effect of the ambient lightingon the adaptation of the human visual system comprises modeling aplurality of adaptation aspects to obtain a corresponding plurality ofestimated adaptations and the method comprises adjusting differentcharacteristics of the mappings based on different ones of the estimatedadaptations.
 9. A method according to claim 8 wherein modeling theeffect of the ambient lighting on light adaptation of the human visualsystem is based at least in part on inputs from one or more ambientlight sensors and on image content displayed on the target display. 10.A method according to claim 8 comprising controlling a luminance towhich the predetermined image value is mapped based at least in part onan estimate of a general brightness adaptation.
 11. A method accordingto claim 8 comprising controlling mid-range contrast based at least inpart on an estimate of lateral brightness adaptation.
 12. A methodaccording to claim 8 comprising controlling contrast in highlight andshadow regions based at least in part on an estimate of local brightnessadaptation.
 13. A method according to claim 1 wherein the target displayhas a greater peak luminance than the reference display and the mappingis configured to expand the luminance corresponding to image valuescorresponding to a highlight region above the mid-tone region.
 14. Amethod according to claim 1 wherein the target display has a lower blacklevel than the reference display and the mapping is configured to expandthe luminance corresponding to image values corresponding to a shadowregion below the mid-tone region.